fix(elasticsearch): skip bootstrap-only checks when joining existing cluster#149
Conversation
…cluster Two checks in the elasticsearch role assume a fresh cluster bootstrap and fail when a node is added to an existing cluster: 1. Master-count check (tasks/main.yml) requires an odd count of master nodes among `ansible_play_hosts_all`. When joining one or two new masters at a time, the limited play has 1-2 master-eligible hosts and the check fires even though the existing cluster already has its own quorum. 2. Runtime cluster setup block (tasks/elasticsearch-security.yml) tries the bootstrap password against the ES API, intending to seed elastic user, built-in user passwords, etc. On a node joining an existing cluster the elastic user already has its real password (replicated via cluster state), so bootstrap auth fails and the block aborts with: "Bootstrap password authentication failed and this is not the CA host". Both fixes gate the relevant tasks on `elasticsearch_cluster_set_up`, consistent with the pattern from #146/#147: when the user signals that the cluster is already bootstrapped, do not run bootstrap-only logic. Closes #148
|
Warning Rate limit exceeded
To continue reviewing without waiting, purchase usage credits in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…xisting cluster (#149)" (#150) This reverts commit 39037c0. The block-level `when: not elasticsearch_cluster_set_up | bool` on the "Runtime cluster setup" block in elasticsearch-security.yml regresses fresh-install scenarios. The block contains an inner set_fact that flips elasticsearch_cluster_set_up to true partway through, after the initial passwords file is written. Block-level when is copied to every inner task and re-evaluated against the current variable state, so every task after that internal set_fact silently skips — including the "Fetch elastic password after initial setup" task that sets elasticstack_password. Downstream consumers (cluster settings, watermarks, the molecule shared set_ci_watermarks helper) then see elasticstack_password undefined and either fall back to a hardcoded file path that doesn't exist on custom-path scenarios, or fail outright. Symptom in CI: elasticsearch_custom, elasticsearch_cert_content, and elasticsearch_diagnostics scenarios fail at the watermark setup step on rocky9/debian13 + ES 9. Previously green on the same scenarios. Reverting both files together. The main.yml master-count change is reverted alongside it because it's only useful in combination with the join-existing-cluster path that needs further work to land cleanly. A follow-up PR will gate only the bootstrap-password tasks individually instead of the whole block. Reopens #148.
…cluster (#151) Adds `not elasticsearch_cluster_set_up | bool` to the four bootstrap-password tasks in elasticsearch-security.yml plus the master-count check in main.yml, so a joining node skips the checks that only make sense for fresh-bootstrap. Reopens issue #148 after the previous attempt (#149, reverted in #150) regressed fresh-install scenarios. The previous attempt put the gate on the whole "Runtime cluster setup" block. That block contains an inner `set_fact` that flips elasticsearch_cluster_set_up to true after the initial passwords file is written. Ansible re-evaluates block-level whens per task against the live variable, so every inner task after that flip silently skipped, including the fetch that sets elasticstack_password. Downstream tasks then saw it undefined and either fell back to a hardcoded path that doesn't exist on custom-path scenarios, or failed outright. Gating per task avoids that — each task's when is independent and doesn't get re-evaluated against later state. The four gated tasks all already had `not elasticsearch_passwords_file.stat.exists | bool` in their when, so they're a clean place for the additional guard. The master-count check in main.yml was uncontroversial in #149 — it sits before the security tasks run and the inner flip never reaches it — so it returns unchanged. No molecule scenario covers the join-existing-cluster path: it would need a multi-node setup where the joining node sees the passwords file as absent (stat delegated to the CA host), which the existing single-node scenarios can't reproduce. Existing scenarios still validate the fresh-bootstrap path and would catch a regression like the one #149 introduced.
Summary
Fixes #148. Builds on the same pattern as #146 / #147 (respect
elasticsearch_cluster_set_up: trueto avoid bootstrap-only behaviour on join-existing-cluster nodes).Two checks in the role would fail when a fresh node joins an existing cluster:
In both cases the joining node is in fact correctly added to the cluster (verifiable via `_cat/nodes`), so these checks are blocking a healthy outcome.
Change
Comments added in both places explaining why.
Test plan
Notes
These three issues (#146, #148) point to a common pattern: the role bundles "first-time security bootstrap" with normal config rendering. Long-term it might be worth splitting them into separate task files, but this minimal fix addresses the immediate blocker.